Imaging devices for capturing color and depth information

ABSTRACT

An imaging device includes a pixel array including a plurality of pixels. Each pixel includes a photoelectric conversion region that converts incident light into electric charge, and a first transfer transistor coupled to a first floating diffusion and the photoelectric conversion region. The imaging device includes a first driving circuit to control the plurality of pixels in an imaging mode to generate a color image, and a second driving circuit to control the plurality of pixels in a depth mode to generate a depth image.

FIELD

Example embodiments are directed to imaging devices, imagingapparatuses, and methods for operating the same, and more particularly,to imaging devices, imaging apparatuses, and methods for capturing colorand depth information.

BACKGROUND

Imaging sensing has applications in many fields, including objecttracking, environment rendering, etc. Some image sensors employtime-of-flight (ToF) principles to detect a distance to an object orobjects within a scene. In general, a ToF depth sensor includes a lightsource and an imaging device including a plurality of pixels for sensingreflected light. In operation, the light source emits light (e.g.,infrared light) toward an object or objects in the scene, and the pixelsdetect the light reflected from the object or objects. The elapsed timebetween the initial emission of the light and receipt of the reflectedlight by each pixel may correspond to a distance from the object orobjects. Direct ToF imaging devices may measure the elapsed time itselfto calculate the distance while indirect ToF imaging devices may measurethe phase delay between the emitted light and the reflected light andtranslate the phase delay into a distance. The depth values of thepixels are then used by the imaging device to determine a distance tothe object or objects, which may be used to create a three dimensionalscene of the captured object or objects.

SUMMARY

Example embodiments relate to imaging devices, imaging apparatuses, andmethods thereof that enable capturing color and depth information usinga same set of pixels.

At least one example embodiment is directed to an imaging deviceincluding a pixel array including a plurality of pixels. Each pixelincludes a photoelectric conversion region that converts incident lightinto electric charge, and a first transfer transistor coupled to a firstfloating diffusion and the photoelectric conversion region. The imagingdevice includes a first driving circuit to control the plurality ofpixels in an imaging mode to generate a color image, and a seconddriving circuit to control the plurality of pixels in a depth mode togenerate a depth image.

According to at least one example embodiment, the imaging deviceincludes a plurality of color filters that correspond to the pluralityof pixels, and the plurality of color filters include red color filters,green color filters, blue color filters, and neutral color filters.

According to at least one example embodiment, the neutral color filtersinclude white color filters, gray color filters, or black color filters.

According to at least one example embodiment, the imaging deviceincludes an optical filter on the plurality of color filters and thatpasses visible light and selected wavelengths of infrared light.

According to at least one example embodiment, the optical filter blockswavelengths of light between a wavelength of the visible light and awavelength of the selected wavelengths of infrared light.

According to at least one example embodiment, the second driving circuitapplies first, second, third, and fourth transfer signals to the firsttransfer transistor in first, second, third, and fourth frames,respectively, to generate a first pixel value for the first frame, asecond pixel value for the second frame, a third pixel value for thethird frame, and a fourth pixel value for the fourth frame. The first,second, third, and fourth pixel values are used to calculate a distanceto an object.

According to at least one example embodiment, the first, second, third,and fourth transfer signals have respective phase shifts of 0 degrees,180 degrees, 90 degrees, and 270 degrees compared to a driving signal ofa light source that emits light toward the object.

According to at least one example embodiment, the first driving circuitcontrols the plurality of pixels to output color data for the colorimage in a fifth frame.

According to at least one example embodiment, the first driving circuitand the second driving circuit control the plurality of pixels through asame set of signal lines.

According to at least one example embodiment, the first driving circuitincludes first switching circuitry to connect the set of signal lines tothe plurality of pixels in the imaging mode and disconnect the set ofsignal lines from the plurality of pixels in the depth mode. The seconddriving circuit includes second switching circuitry to connect the setof signal lines to the plurality of pixels in the depth mode and todisconnect the set of signal lines from the plurality of pixels in theimaging mode.

According to at least one example embodiment, each pixel furthercomprises a second transfer transistor coupled to a second floatingdiffusion and the photoelectric conversion region.

According to at least one example embodiment, the second driving circuitapplies a first transfer signal to the first transfer transistor of afirst pixel during a first frame to generate a first pixel value,applies a second transfer signal to the second transfer transistor ofthe first pixel during the first frame to generate a second pixel value,applies a third transfer signal to the first transfer transistor of asecond pixel during the first frame to generate a third pixel value, andapplies a fourth transfer signal to the second transfer transistor ofthe second pixel during the first frame to generate a fourth pixelvalue. The first, second, third, and fourth pixel values are used tocalculate a distance to an object.

According to at least one example embodiment, the first driving circuitcontrols the plurality of pixels to output color data for the colorimage in a second frame.

According to at least one example embodiment, the first, second, third,and fourth transfer signals have respective phase shifts of 0 degrees,180 degrees, 90 degrees, and 270 degrees compared to a driving signal ofa light source that emits light toward the object.

According to at least one example embodiment, the second driving circuitapplies the second transfer signal to the first transfer transistor ofthe first pixel during a second frame to generate a fifth pixel value,applies the first transfer signal to the second transfer transistor ofthe first pixel during the second frame to generate a sixth pixel value,applies the fourth transfer signal to the first transfer transistor ofthe second pixel during the second frame to generate a seventh pixelvalue, and applies the third transfer signal to the second transfertransistor of the second pixel during the second frame to generate aneighth pixel value.

According to at least one example embodiment, the first, second, third,fourth, fifth, sixth, seventh, and eighth pixel values are used tocancel fixed pattern noise in a distance calculation to the object.

According to at least one example embodiment, the first driving circuitand the second driving circuit control the plurality of pixels through asame set of signal lines.

According to at least one example embodiment, the first driving circuitcontrols the plurality of pixels to output color data for the colorimage in a third frame.

At least one example embodiment is directed to a system including alight source that emits infrared light, and an imaging device thatincludes a pixel array including a plurality of pixels. Each pixelincludes a photoelectric conversion region that converts incident lightinto electric charge, and a first transfer transistor coupled to a firstfloating diffusion and the photoelectric conversion region. The imagingdevice includes a first driving circuit to control the plurality ofpixels in an imaging mode to generate a color image based on visiblelight received from a scene, and a second driving circuit to control theplurality of pixels in a depth mode to generate a depth image based onthe infrared light reflected from the scene.

At least one example embodiment is directed to a method that includesdriving, by a first driving circuit, a plurality of pixels in an imagingmode to generate a color image, and driving, by a second drivingcircuit, the plurality of pixels in a depth mode to generate a depthimage. The first driving circuit and the second driving circuit drivethe plurality of pixels through a same set of signal lines.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an imaging device according to at least oneexample embodiment.

FIG. 2 illustrates an example schematic of a pixel from FIG. 1 accordingto at least one example embodiment.

FIG. 3 illustrates an example pixel array having a color filter array(CFA) used to sense color information and depth information according toat least one example embodiment.

FIG. 4 illustrates an example diagram for capturing depth and colorinformation using the CFA of FIG. 3 according to at least one exampleembodiment.

FIG. 5 illustrates example characteristics of an imaging device thatincludes the CFA of FIG. 3 according to at least one example embodiment.

FIG. 6 illustrates another example of a CFA according to at least oneexample embodiment.

FIG. 7 illustrates an example readout method for collecting colorinformation and depth information according to at least one exampleembodiment.

FIG. 8 illustrates an example schematic of a pixel array for achievingthe method of FIG. 7 according to at least one example embodiment.

FIG. 9 illustrates an example wiring layout for achieving the method ofFIG. 7 according to at least one example embodiment.

FIG. 10 illustrates another example wiring layout for achieving themethod of FIG. 7 according to at least one example embodiment.

FIG. 11 illustrates an example readout method for collecting color anddepth information according to at least one example embodiment.

FIG. 12 illustrates further details of the example readout method inFIG. 11 according to at least one example embodiment.

FIG. 13 illustrates an example schematic for achieving the method ofFIGS. 11 and 12 according to at least one example embodiment.

FIG. 14 illustrates an example wiring layout for the schematic in FIG.13 according to at least one example embodiment.

FIG. 15 illustrates an example wiring layout for the schematic in FIG.13 according to at least one example embodiment.

FIG. 16 illustrates an example read out method according to at least oneexample embodiment.

FIG. 17 illustrates further details of the example read out method inFIG. 16 according to at least one example embodiment.

FIG. 18 illustrates an example schematic for achieving the examplemethod in FIGS. 16 and 17 according to at least one example embodiment.

FIG. 19 illustrates an example wiring layout for the schematic in FIG.18 according to at least one example embodiment.

FIG. 20 illustrates an example wiring layout for the schematic in FIG.18 according to at least one example embodiment.

FIG. 21 illustrates an example read out method according to at least oneexample embodiment.

FIG. 22 illustrates example circuitry and timing diagram for driving alight source that produces the reference optical signal used forcollecting depth information according to at least one exampleembodiment.

FIG. 23 illustrates an example structure of a pixel array that includespixels and an optical filter according to at least one exampleembodiment.

FIG. 24 illustrates example processing operations for removing infraredlight during color processing of a color image obtain during an imagingmode according to at least one example embodiment.

FIG. 25 illustrates example equations for cancelling FPN offsetsaccording to at least one example embodiment.

FIG. 26 is a block diagram illustrating an example of a ranging modulewith the ability to capture color information according to at least oneexample embodiment.

FIG. 27 is a diagram illustrating use examples of an imaging deviceaccording to at least one example embodiment.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of an imaging device according to at least oneexample embodiment.

The pixel 51 includes a photoelectric conversion region PD, such as aphotodiode or other light sensor, transfer transistors TG0 and TG1,floating diffusion regions FD0 and FD1, reset transistors RST0 and RST1,amplification transistors AMP0 and AMP1, and selection transistors SEL0and SEL1.

The imaging device 1 shown in FIG. 1 may be an imaging sensor of a frontor rear surface irradiation type, and is provided, for example, in animaging apparatus having a ranging function (or distance measuringfunction).

The imaging device 1 has a pixel array unit (or pixel array or pixelsection) 20 formed on a semiconductor substrate (not shown) and aperipheral circuit integrated on the same semiconductor substrate thesame as the pixel array unit 20. The peripheral circuit includes, forexample, a tap driving unit (or tap driver) 21, a vertical driving unit(or vertical driver) 22, a column processing unit (or column processingcircuit) 23, a horizontal driving unit (or horizontal driver) 24, and asystem control unit (or system controller) 25.

The imaging device element 1 is further provided with a signalprocessing unit (or signal processor) 31 and a data storage unit (ordata storage or memory or computer readable storage medium) 32. Notethat the signal processing unit 31 and the data storage unit 32 may bemounted on the same substrate as the imaging device 1 or may be disposedon a substrate separate from the imaging device 1 in the imagingapparatus.

The pixel array unit 20 has a configuration in which pixels 51 thatgenerate charge corresponding to a received light amount and output asignal corresponding to the charge are two-dimensionally disposed in amatrix shape of a row direction and a column direction. That is, thepixel array unit 20 has a plurality of pixels 51 that performphotoelectric conversion on incident light and output a signalcorresponding to charge obtained as a result. Here, the row directionrefers to an arrangement direction of the pixels 51 in a horizontaldirection, and the column direction refers to the arrangement directionof the pixels 51 in a vertical direction. The row direction is ahorizontal direction in the figure, and the column direction is avertical direction in the figure.

The pixel 51 receives light incident from the external environment, forexample, infrared light, performs photoelectric conversion on thereceived light, and outputs a pixel signal according to charge obtainedas a result. The pixel 51 may include a first charge collector thatdetects charge obtained by the photoelectric conversion PD by applying apredetermined voltage (first voltage) to the pixel 51, and a secondcharge collector that detects charge obtained by the photoelectricconversion by applying a predetermined voltage (second voltage) to thepixel 51. The first and second charge collector may include tap A andtap B, respectively. Although two charge collectors are shown (i.e., tapA, and tap B), more or fewer charge collectors may be included accordingto design preferences. The first voltage and the second voltage assistwith channeling charge toward tap A and tap B during different timeperiods. The charge is then read out of each tap A and B with transfersignals, discussed in more detail below.

The tap driving unit 21 supplies the predetermined first voltage to thefirst charge collector of each of the pixels 51 of the pixel array unit20 through a predetermined voltage supply line 30, and supplies thepredetermined second voltage to the second charge collector thereofthrough the predetermined voltage supply line 30. Therefore, two voltagesupply lines 30 including the voltage supply line 30 that transmits thefirst voltage and the voltage supply line 30 that transmits the secondvoltage are wired to one pixel column of the pixel array unit 20.

In the pixel array unit 20, with respect to the pixel array of thematrix shape, a pixel drive line 28 is wired along a row direction foreach pixel row, and two vertical signal lines 29 are wired along acolumn direction for each pixel column. For example, the pixel driveline 28 transmits a drive signal for driving when reading a signal fromthe pixel. Note that, although FIG. 1 shows one wire for the pixel driveline 28, the pixel drive line 28 is not limited to one. One end of thepixel drive line 28 is connected to an output end corresponding to eachrow of the vertical driving unit 22.

The vertical driving unit 22 includes a shift register, an addressdecoder, or the like. The vertical driving unit 22 drives each pixel ofall pixels of the pixel array unit 20 at the same time, or in row units,or the like. That is, the vertical driving unit 22 includes a drivingunit that controls operation of each pixel of the pixel array unit 20,together with the system control unit 25 that controls the verticaldriving unit 22.

The signals output from each pixel 51 of a pixel row in response todrive control by the vertical driving unit 22 are input to the columnprocessing unit 23 through the vertical signal line 29. The columnprocessing unit 23 performs a predetermined signal process on the pixelsignal output from each pixel 51 through the vertical signal line 29 andtemporarily holds the pixel signal after the signal process.

Specifically, the column processing unit 23 performs a noise removalprocess, a sample and hold (S/H) process, an analog to digital (AD)conversion process, and the like as the signal process.

The horizontal driving unit 24 includes a shift register, an addressdecoder, or the like, and sequentially selects unit circuitscorresponding to pixel columns of the column processing unit 23. Thecolumn processing unit 23 sequentially outputs the pixel signalsobtained through the signal process for each unit circuit, by aselective scan by the horizontal driving unit 24.

The system control unit 25 includes a timing generator or the like thatgenerates various timing signals and performs drive control on the tapdriving unit 21, the vertical driving unit 22, the column processingunit 23, the horizontal driving unit 24, and the like, on the basis ofthe various generated timing signals.

The signal processing unit 31 has at least a calculation processfunction and performs various signal processing such as a calculationprocess on the basis of the pixel signal output from the columnprocessing unit 23. The data storage unit 32 temporarily stores datanecessary for the signal processing in the signal processing unit 31.The signal processing unit 31 may control overall functions of theimaging device 1. For example, the tap driving unit 21, the verticaldriving unit 22, the column processing unit 23, the horizontal drivingunit 24, and the system control unit 25, and the data storage unit 32may be under control of the signal processing unit 31. The signalprocessing unit or signal processor 31, alone or in conjunction with theother elements of FIG. 1, may control all operations of the systemsdiscussed in more detail below with reference to the accompanyingfigures. Thus, the terms “signal processing unit” and “signal processor”may also refer to a collection of elements 21, 22, 23, 24, 25, and/or31. A signal processor according to at least one example embodiment iscapable of processing color information to produce a color informationand depth information to produce a depth image.

FIG. 2 illustrates an example schematic of a pixel 51 from FIG. 1. Thepixel 51 includes a photoelectric conversion region PD, such as aphotodiode or other light sensor, transfer transistors TG0 and TG1,floating diffusion regions FD0 and FD1, reset transistors RST0 and RST1,amplification transistors AMP0 and AMP1, and selection transistors SEL0and SEL1. The pixel 51 may further include an overflow transistor OFG,transfer transistors FDG0 and FDG1, and floating diffusion regions FD2and FD3.

The pixel 51 may be driven according to control signals or transfersignals GD0, GD90, GD180 and GD270 applied to gates or taps A/B oftransfer transistors TG0/TG1, reset signal RSTDRAIN, overflow signalOFGn, power supply signal VDD, selection signal SELn, and verticalselection signals VSL0 and VSL1. These signals are provided by variouselements from FIG. 1, for example, the tap driver 21, vertical driver22, system controller 25, etc.

As shown in FIG. 2, the transfer transistors TG0 and TG1 are coupled tothe photoelectric conversion region PD and have taps A/B that transfercharge as a result of applying transfer signals.

These transfer signals GD0, GD90, GD180, and GD270 may have differentphases relative to a phase of a modulated signal from a light source(e.g., phases that differ 0 degrees, 90 degrees, 180 degrees, and/or 270degrees). The transfer signals may be applied in a manner that allowsfor depth information (or pixel values) to be captured in a desirednumber of frames (e.g., one frame, two frames, four frames, etc.). Oneof ordinary skill in the art would understand how to apply the transfersignals in order to use the collected charge to calculate a distance toan object. In at least one example embodiment, other transfer signalsmay be applied in a manner that allows for color information to becaptured for a color image.

It should be appreciated that the transfer transistors FDG0/FDG1 andfloating diffusions FD2/FD3 are included to expand the charge capacityof the pixel 51, if desired. However, these elements may be omitted ornot used, if desired. The overflow transistor OFG is included totransfer overflow charge from the photoelectric conversion region PD,but may be omitted or unused if desired. Further still, if only one tapis desired, then elements associated with the other tap may be unused oromitted (e.g., TG1, FD1, FDG1, RST1, SEL1, AMP1).

It should be understood that figures depicting pixel layouts discussedbelow show substantially accurate relative positional relationships ofthe elements depicted therein and can be relied upon as support for suchpositional relationships. For example, the figures provide support forselection transistors SEL and amplification transistors AMP beingaligned with one another in a vertical direction. As another example,the figures provide support for an element on a right side of a figurebeing aligned with an element on a left side of a figure in thehorizontal direction. As yet another example, the figures are generallyaccurate with respect to showing positions of overlapping elements.

In addition, where reference to general element or set of elements isappropriate instead of a specific element, the description may refer tothe element or set of elements by its root term. For example, whenreference to a specific transfer transistor TG0 or TG1 is not necessary,the description may refer to the transfer transistor(s) “TG.”

FIGS. 3-5 illustrate inventive concepts according to at least oneexample embodiment. In more detail, FIG. 3 illustrates an example pixelarray 300 having a color filter array (CFA) used to sense colorinformation and depth information. Each pixel in the pixel array maycorrespond to one of the pixels 51 above. As shown, the CFA uses red R,green G, and blue B color filters in a Bayer pattern, except that asubset of green color filters in the original Bayer pattern are neutralN (e.g., white) to detect infrared light to allow for a method thatenables capture of color information and depth information by the pixelarray. In order to allow for detection of infrared (IR) light, pixelswith red, green, and blue color filters do not include an IR cut filter.

FIG. 4 illustrates an example diagram for capturing depth and colorinformation using the CFA of FIG. 3. As shown for frames 1 and 2, areference optical signal (e.g., modulated infrared IR light) may beemitted toward an object, and the reflected (IR) light signal causecharges to be generated in the photodiodes, where the charges are thentransferred from respective photoelectric conversion regions of thepixels 51 to floating diffusions FD0/FD1 according to transfer signalsGDA, GDB, GDC, GDD (e.g., applied to transfer transistors in the pixels)having the phases shown with respect to the reference optical signal.Throughout this description, GDA, GDB, GDC, and GDD correspond to GD0,GD90, GD180, and GD270 form FIG. 2, respectively. In frames 1 and 2, thetransfer signals may be applied to taps (e.g., gates of transfertransistors) of pixels to transfer charge from respective photoelectricconversion regions, where the transfer signals are phase shifted 0, 90,180, and 270 degrees from the reference optical signal. For example, inFrame 1, for pixels 51 with two taps which are identified by taps A andB, pixel signals or pixel values p0 and p90 may be associated with tapA, whereas pixel values p180 and p270 may be associated with tap B. InFrame 2, the transfer signals maybe applied to the taps of the pixels,where the transfer signals are phase shifted 180, 0, 270, and 90 degreesfrom the reference optical signal. For example, pixel values p180′ andp270′ may be associated with tap A, whereas pixel values p0′ and p90′may be associated with tap B. FIGS. 16 and 17 describe FIG. 4 in moredetail. In Frame 3, IR illumination is terminated and RGB data is readout in accordance with known techniques for the purpose of producing acolor image.

FIG. 5 illustrates example characteristics of an imaging device 1 thatincludes the CFA 300 of FIG. 3. As shown, an IR notch pass opticalfilter may be used in conjunction with the CFA 300 to pass most visiblelight, block certain wavelengths of light in the visible and IRspectrums, and pass certain wavelengths of IR light (see also FIG. 23).

FIG. 6 illustrates another example of a CFA 600 according to at leastone example embodiment. The CFA 600 of FIG. 6 is a Bayer pattern exceptthat a subset of the green color filters N in the original Bayer patternare black or other neutral color (e.g., a shade of gray) that passesinfrared light (e.g., due to reflections of the reference optical signalfrom an object). Although not explicitly shown, it should be understoodthat each color filter in the CFA 600 is associated with a pixelincluding a photoelectric conversion region and a plurality oftransistors for reading out electric charge (e.g., transfer transistors,overflow transistors, selection transistors, amplification transistors,etc.). In addition, it should be understood that each color filter inthe CFAs 300/600 shown in FIGS. 3 and 6 may be further divided intosub-filters that correspond to sub-pixels. For example, each colorfilter block may be divided into four, eight, or more, sub-blocks tofurther improve resolution of the imaging device 1.

FIG. 7 illustrates an example readout method for collecting colorinformation and depth information. As shown, Frames 1-4 may be used forreading out depth information by reading out electric charge as pixelvalues p0, p180, p90, p270 collected at 0, 180, 90, and 270 degreesphase shifts from the reference optical signal while Frame 5 is used toread out RGB color information. Each frame may comprise a desired numberof modulation cycles where, for each modulation cycle, the light sourceemits a light signal and charge is detected with a transfer signal. Thefinal pixel value (e.g., p0) for a particular phase may be the totalamount of charge collecting for all modulation cycles in that frame.FIG. 7 illustrates an embodiment where only one tap per pixel is used tocollect depth and color information. Accordingly, a pixel arrayconfigured to operate in accordance with FIG. 7 may not have the two tapper pixel configuration described with reference to FIGS. 1, 2 and 4, orone tap may be unused. Frames 1 thru 4 and 5 may be consecutive framesor frames may be skipped between each frame 1 thru 5 if desired.

FIGS. 8-10 illustrate example structures for achieving the method ofFIG. 7. As shown, the pixel array 800 in FIG. 8 may employ two drivers,an imaging driver (or driving circuit) 810 for driving the pixels 51 tocollect color information in an imaging frame(s) and a depth driver (ordriving circuit 815) for driving the pixels 51 to collect depthinformation in a depth frame(s). These drivers may be included in orseparate from elements in FIG. 1. To collect color information, theimaging driver 810 may employ row by row control (row 3, row 2, row 1,row 0), while to collect depth information, the depth driver 815 mayemploy global control by applying transfer signals 0, 90, 180, and 270degrees phase shifted from a light signal. FIG. 8 illustrates two groupsof four blocks where each block represents a pixel. Each block islabeled with that pixel's associated phases 0/180 and 90/270. Thenotation 0/180 indicates that tap A of a pixel receives a transfersignal with 0 degrees phase difference from the light signal while tap Breceives a transfer signal with 180 degrees phase difference from thelight signal. The same is true for the notation 90/270 except thetransfer signals are 90 degrees phase shifted and 270 degrees phaseshifted. In general, each pixel 51 in FIG. 8 has the same or similarstructure as the pixel of FIG. 2. FIG. 8 further illustrates varioussignal lines connected to the elements of each pixel. These signal linesinclude reset signal lines RST[0, 1, 2, 3,], vertical signal linesVSL[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], transfer signal lines FDG [0, 1, 2,3], transfer signal lines GDA[0], GDB[0] (with connections GD_Odd[0] topixels in odd row numbers and GD_Even[0] to pixels in even row numbers),power signal lines VDDHPX and RSTDRAIN, ground signal lines GND toground an unused tap (tap B in this example), and signal lines OFGconnected to gates of overflow transistors OFG. In an imaging mode,imaging driver 810 may apply signals to these signal lines, while in adepth mode, the depth driver 815 may apply signals to the signal lines.

FIG. 9 illustrates an example wiring layout 900 where one control linedrives transfer transistors in two rows. The photoelectric conversionregions PD are denoted by the octagonal shapes, and connections totransfer transistors TG0/TG1 are indicated by taps A and B. FIG. 9 showsswitches 905 and 910 (which may be included in the drivers 810 and 815,respectively) for switching between an imaging mode and a depth mode atouter regions of the layout 900, wirings W, and connections C to wiringsW. As shown, the wirings W connect signal lines SL (which correspond tosignal lines from FIG. 8) to gates or taps AB of transistors TG0/TG1.The wirings W and connections C in FIG. 9 may be formed in a wiringlayer of the imaging device (e.g., an M3 wiring layer), while the signallines SL are formed in a different wiring layer. FIG. 9 furtherillustrates unlabeled transistors which correspond to transistors fromFIG. 2. The photoelectric conversion regions PD, signal lines SL,wirings W, connections C, and transistors have the shown relativepositional relationships. In general, the signal lines SL extend in afirst direction (e.g., a horizontal direction) and are at arranged atregular intervals while the wirings W include portions that extend inthe first direction and portions that extend in a second directionperpendicular to the first direction (e.g., a vertical direction).

To collect color information, only one of the transfer gates (e.g. TG0)or taps (A) is used, and the other transfer gate (e.g., TG1) or tap (B)is grounded with GND. In other words, a pixel in the imaging mode workssimilar to a pixel with a single transfer gate. However, exampleembodiments are not limited thereto, and the roles of TG0 and TG1 may bereversed if desired. That is, TG1 may be used to transfer signal in theimaging mode while TG0 is kept off. In any event, it should beunderstood that only one of the transfer transistors for each pixel 51is used for transferring charge for color sensing.

To collect depth information, the odd rows may receive transfer signalsat taps B and the even rows may receive transfer signals at taps A.

The transfer signals for collecting color and depth information may thenbe applied in accordance with FIG. 7. For example, in a first frame forcharge transfer, the transfer signals applied to taps A may have a phaseshift of 0 degrees compared to the reference optical signal. In a secondframe for charge transfer, the transfer signals applied to taps A mayhave a phase shift of 180 degrees compared to the reference opticalsignal. In a third frame for charge transfer, the transfer signalsapplied to taps A may have a phase shift of 90 degrees compared to thereference optical signal. In a fourth frame, the transfer signalsapplied to taps A may have a phase shift of 270 degrees compared to thereference optical signal. In these four frames, tap B is pulsed with asignal having a 180 degree phase shift with respect to tap A. Forexample, in FIG. 8, if tap A is at 0 degrees, then tap B is at 180degrees in the same frame; and if tap A is at 270 degrees, tap B is at90 degrees in the same frame. In a fifth frame, the depth driver 815 isdeactivated and the imaging driver 810 is activated to transfer chargeused for generating color information by applying signals to signallines GD_Even and GD_Odd. Thus, the charge collected by each FD isreadout according to the diagram of FIG. 7.

FIG. 10 illustrates another example wiring layout 1000 for achieving thereadout method of FIG. 7. In FIG. 10, a single signal line SL drives onerow of pixels 51, and the pixels 51 may be driven according the diagramof FIG. 7 as explained above with reference to FIG. 9. In FIG. 10, thephotoelectric conversion regions PD, signal lines SL, wirings W,connections C, and transistors have the shown relative positionalrelationships. The signal lines SL may be arranged at regular intervalsin two groups of four (i.e., a top group and a bottom group).

FIGS. 11 and 12 illustrate an example readout method for collectingcolor and depth information according to at least one exampleembodiment.

As shown in FIGS. 11 and 12, charge collected according to all fourtransfer signals is read out in a first frame while charge collected forcolor information is read out in a second frame. For example, inoperation, pixel (0,0) has two taps A and B that transfer chargeaccording to signals that are 0 and 180 degrees out of phase from thereference optical signal, while pixel (1,0) has two taps A and B thattransfer charge according to signals that are 90 and 270 degrees out ofphase with the reference optical signal. Pixel (0,1) is driven the sameas pixel (0,0) and pixel (1,1) is driven the same as pixel (1,0). Thisallows a group of two pixels to collect charge as pixel values p0, p90,p180, and p270 for phases 0, 90, 180 and 270, which would be sufficientto do depth calculations in one frame. Although not explicitly shown, itshould be understood that in another embodiment two phases may be readout in a first frame, two phases may be read out in a second frame, andthe color information may be read out in a third frame.

FIG. 13 illustrates an example schematic 1300 for achieving the methodof FIGS. 11 and 12 and FIGS. 14-15 illustrates example wiring layoutsfor achieving the method of FIGS. 11 and 12. FIG. 13 illustrates aschematic having two drivers (or driving circuits) 1305 and 1310 asnoted above with reference to FIG. 8. FIG. 13 includes many of the sameelements as FIG. 8, and thus a description of these elements is notrepeated. Compared to FIG. 8, FIG. 13 further includes signal linesGDC[0] and GDD[0] in order to carry out the method of FIGS. 11 and 12.

FIG. 14 illustrates an example wiring layout 1400 where one signal lineSL controls two rows of pixels 51. To collect depth information, signallines GND, GD_Even, and GD_Odd are driven in the same manner as noteabove in the description of FIG. 9. Meanwhile, to collect depthinformation, signal lines GDA, GDB, GDC, and GDD receive differenttransfer signals with different phases. For example, signal lines GDA,GDB, GDC, and GDD receive signals having 0, 180, 90, and 270 degreesphase shifts, respectively, compared to a reference optical signal. FIG.14 includes switches 1405 and 1410, which are on or off depending onwhether the imaging device is in a depth mode or an imaging mode. Eachswitch 1405/1410 may be included in a respective driving circuit1305/1310. In FIG. 14, the photoelectric conversion regions PD, signallines SL, wirings W, connections C, and transistors have the shownrelative positional relationships. The signal lines SL may be arrangedat regular intervals.

FIG. 15 illustrates an example wiring layout 1500 where one control linedrives one row of pixels. In FIG. 15, the photoelectric conversionregions PD, signal lines SL, wirings W, connections C, and transistorshave the shown relative positional relationships. The signal lines SLmay be arranged at regular intervals in two groups of four (i.e., a topgroup and a bottom group).

FIGS. 16 and 17 illustrate an example read out method according to atleast one example embodiment. A first Frame 1 may be the same as thefirst frame of FIG. 12 while in a second Frame 2 phases for taps A and Bof the pixels 51 are inverted to collect pixel values p180′, p0′,p2′70′, and p90′. This method allows for cancellation of fixed patternnoise (FPN) offsets. Color information may be read out in a third Frame3.

FIG. 18 illustrates a schematic 1800 for achieving the example method ofFIGS. 16 and 17 while FIGS. 19 and 20 illustrate example wiring layouts1900 and 2000 for the same. As in FIGS. 8 and 13, FIG. 18 shows animaging driver 1805 for controlling readout of color information and adepth driver 1810 for controlling readout of depth information. FIG. 18includes the same elements as FIG. 13, and thus a description of theseelements is not repeated here.

As shown in FIG. 19, one signal line SL drives two pixel rows. As shownin FIG. 20, one signal line drives one row of pixels. To collect depthinformation, transfer signals are applied to the signal lines GDA, GDB,GDC, and GDD in a manner consistent with the method of FIGS. 16 and 17.To collect color information, transfer signals are applied to signallines GND, GD_Even, and GD_Odd in the same manner as described abovewith reference to FIGS. 9, 10, and 14, and 15. FIG. 19 includes switches1905 and 1910, which are on or off depending on whether the imagingdevice is in an imaging mode or a depth mode. In FIG. 19, thephotoelectric conversion regions PD, signal lines SL, wirings W,connections C, and transistors have the shown relative positionalrelationships. The signal lines SL may be arranged at regular intervals.

In FIG. 20, the photoelectric conversion regions PD, signal lines SL,wirings W, connections C, and transistors have the shown relativepositional relationships. The signal lines SL may be arranged at regularintervals in two groups of four (i.e., a top group and a bottom group).

FIG. 21 illustrates an example read out method according to at least oneexample embodiment. FIG. 21 is the same as FIG. 17 except that FIG. 21illustrates reading out P-phase and D-phase color data in third andfourth frames, respectively. Here, the P-phase may correspond to a framewhen charge is collected during a reset operation in which thephotoelectric conversion regions PD are reset, and the D-phase maycorrespond to a frame when charge is collected during an exposure periodof the photoelectric conversion regions PD. The method of FIG. 21 may becarried out with the structures in FIGS. 18-20.

FIG. 22 illustrates example circuitry 2200 and timing diagram 2250 fordriving a light source that produces the reference optical signal usedfor collecting depth information. As shown, circuitry 2200 may includethe imaging device 1 (image sensor), a logic element 2205 (e.g., ANDgate), an amplifier 2210, and a light source 2215. In operation, theimaging device 1 sends a modulated signal fmod and a selection signalTOF select to the logic element 2205 (and enter a depth mode) so that adrive signal of the logic element 2205 is fed to an amplifier whichoperates the light source 2215 accordingly. The timing diagram of FIG.22 may be associated with example embodiments described with referenceto FIGS. 7-10. In FIG. 22, the vertical synchronization signal controlsthe beginning and end of each frame.

FIG. 23 illustrates an example structure 2300 of a pixel array thatincludes pixels 51, corresponding color filters R, G, B, N and anoptical filter 2305 that provides the filtering characteristics shown inthe graph 2350. As shown, the optical filter 2350 passes wavelengths ofvisible and selected wavelengths of infrared light while blocking asection of wavelengths in between. The wavelengths of light emitted fromthe light source 2215 are selected to match the selected wavelengths oflight passed by the optical filter 2305.

FIG. 24 illustrates example processing operations for removing infraredlight during color processing of a color image obtain during an imagingmode. For example, FIG. 24 illustrates a graph 2400 that shows spectraldata collected for R, G, B, and N pixels that includes IR light whilegraph 2410 shows desired spectral data with IR light removed. In FIG.24, the neutral N pixel has a white color filter. FIG. 24 shows anexample resultant matrix 2405 that is used for removing infrared lightfrom the collected spectral data to arrive at the desired spectral data.Here, it should be appreciated that the matrix 2405 may vary accordingto the collected and desired spectral data. That is, given collectedspectral data X and desired spectral data Y, the matrix 2405 isdetermined by minimizing a mean square error (MSE) of Y−X over a rangeof wavelengths.

FIG. 25 illustrates example operations for cancelling FPN offsets duringdepth processing of a depth mode according to at least one exampleembodiment (e.g., for the read out methods of FIGS. 17 and 21). Here,the FPN offsets are represented as β0, β1, β2, and β3 while p0, p90,p180, and so on are pixel values associated with a particular phase.Further, α0, α1, α2, and α3 are fixed and/or variable values (e.g.,caused by external conditions such as ambient light) that impact thepixel values. Difference signals are d0, d1, d0′, and d1′, which aredifferences between the shown pixel values. Upon combining differencesignals d0 and d0′, and d1 and d1′, FPN offsets are cancelled. After FPNoffsets are cancelled, the system may calculate a distance to an objectusing known methods (e.g., the arctangent method, two-four pulse ratiomethod, etc.). The arctangent set forth below with Equation (1):

$\begin{matrix}{{{Distance} = {\frac{{C \cdot \Delta}T}{2} = \frac{C \cdot \alpha}{4\pi f_{mod}}}}{\alpha = {\arctan\left( \frac{\phi_{1} - \phi_{3}}{\phi_{0} - \phi_{2}} \right)}}} & (1)\end{matrix}$

Here, C is the speed of light, ΔT is the time delay, fmod is themodulation frequency of the emitted light, φ0 to φ3 are the signalvalues detected with transfer signals having phase differences from theemitted light 0 degrees, 90 degrees, 180 degrees, and 270 degrees,respectively.

Systems/devices that may incorporate the above described imaging deviceswill now be described.

FIG. 26 is a block diagram illustrating an example of a ranging modulewith the ability to capture color information according to at least oneexample embodiment.

The ranging module 5000 includes a light emitting unit 5011, a lightemission control unit 5012, and a light receiving unit 5013.

The light emitting unit 5011 has a light source that emits light havinga predetermined wavelength, and irradiates the object with irradiationlight of which brightness periodically changes. For example, the lightemitting unit 5011 has a light emitting diode that emits infrared lighthaving a wavelength in a range of 780 nm to 1000 nm as a light source,and generates the irradiation light in synchronization with a lightemission control signal CLKp of a rectangular wave supplied from thelight emission control unit 5012.

Note that, the light emission control signal CLKp is not limited to therectangular wave as long as the control signal CLKp is a periodicsignal. For example, the light emission control signal CLKp may be asine wave.

The light emission control unit 5012 supplies the light emission controlsignal CLKp to the light emitting unit 5011 and the light receiving unit5013 and controls an irradiation timing of the irradiation light. Afrequency of the light emission control signal CLKp is, for example, 20megahertz (MHz). Note that, the frequency of the light emission controlsignal CLKp is not limited to 20 megahertz (MHz), and may be 5 megahertz(MHz) or the like.

The light receiving unit 5013 receives reflected light reflected fromthe object, calculates the distance information for each pixel accordingto a light reception result, generates a depth image in which thedistance to the object is represented by a gradation value for eachpixel, and outputs the depth image.

The above-described imaging device 1 is used for the light receivingunit 5013, and for example, the imaging device 1 serving as the lightreceiving unit 5013 generates color images in an imaging mode andcalculates the distance information for each pixel from a signalintensity detected by at least one of taps AB in a depth mode, on thebasis of the light emission control signal CLKp.

As described above, the imaging device 1 shown in FIG. 1 is able to beincorporated as the light receiving unit 5013 of the ranging module 5000that obtains and outputs the information associated with the distance tothe subject by the indirect ToF method. By adopting the imaging device 1of one or more of the embodiments described above, it is possible toimprove one or more distance measurement characteristics of the rangingmodule 5000 (e.g., distance accuracy, speed of measurement, and/or thelike).

FIG. 27 is a diagram illustrating use examples of an imaging device 1according to at least one example embodiment.

For example, the above-described imaging device 1 (image sensor) can beused in various cases of sensing light such as visible light, infraredlight, ultraviolet light, and X-rays as described below. The imagingdevice 1 may be included in apparatuses such as a digital still cameraand a portable device with a camera function which capture images,apparatuses for traffic such as an in-vehicle sensor that capturesimages of a vehicle to enable automatic stopping, recognition of adriver state, measuring distance, and the like. The imaging device 1 maybe included in apparatuses for home appliances such as a TV, arefrigerator, and an air-conditioner in order to photograph a gesture ofa user and to perform an apparatus operation in accordance with thegesture. The imaging device 1 may be included in apparatuses for medicalor health care such as an endoscope and an apparatus that performsangiography through reception of infrared light. The imaging device 1may be included in apparatuses for security such as a securitymonitoring camera and a personal authentication camera. The imagingdevice 1 may be included in an apparatus for beauty such as a skinmeasuring device that photographs skin. The imaging device 1 may beincluded in apparatuses for sports such as an action camera, a wearablecamera for sports, and the like. The imaging device 1 may be included inapparatuses for agriculture such as a camera for monitoring a state of afarm or crop.

In view of the above, it should be appreciated that example embodimentsprovide the ability to capture both color and depth information using asame set of pixels. Example embodiments further provide for multiplereadout methods to capture depth and color information in a desirednumber of frames, and methods for FPN cancellation and removal of IRsignals from color information.

In view of FIGS. 1-27, at least one example embodiment is directed to animaging device 1 including a pixel array including a plurality of pixels51. Each pixel 51 includes a photoelectric conversion region PD thatconverts incident light into electric charge, and a first transfertransistor TG0 coupled to a first floating diffusion FD0 and thephotoelectric conversion region PD. The imaging device 1 includes afirst driving circuit 810/1305/1805 to control the plurality of pixels51 in an imaging mode to generate a color image, and a second drivingcircuit 815/1310/1810 to control the plurality of pixels 51 in a depthmode to generate a depth image.

According to at least one example embodiment, the imaging deviceincludes a plurality of color filters that correspond to the pluralityof pixels 51, and the plurality of color filters include red colorfilters R, green color filters G, blue color filters B, and neutralcolor filters N.

According to at least one example embodiment, the neutral color filtersN include white color filters, gray color filters, or black colorfilters.

According to at least one example embodiment, the imaging device 1includes an optical filter 2305 on the plurality of color filters thatpasses visible light and selected wavelengths of infrared light.

According to at least one example embodiment, the optical filter 2305blocks wavelengths of light between a wavelength of the visible lightand a wavelength of the selected wavelengths of infrared light (see FIG.23).

According to at least one example embodiment, the second driving circuitapplies first, second, third, and fourth transfer signals GD0, GD180,GD90, and GD270 to the first transfer transistor TG0 in first, second,third, and fourth frames, respectively, to generate a first pixel valuep0 for the first frame, a second pixel value p180 for the second frame,a third pixel value p90 for the third frame, and a fourth pixel valuep270 for the fourth frame. The first, second, third, and fourth pixelvalues are used to calculate a distance to an object.

According to at least one example embodiment, the first, second, third,and fourth transfer signals have respective phase shifts of 0 degrees,180 degrees, 90 degrees, and 270 degrees compared to a driving signal ofa light source that emits light toward the object.

According to at least one example embodiment, the first driving circuitcontrols the plurality of pixels to output color data for the colorimage in a fifth frame (see FIG. 7, for example).

According to at least one example embodiment, the first driving circuitand the second driving circuit control the plurality of pixels 51through a same set of signal lines SL (see FIG. 9, for example).

According to at least one example embodiment, the first driving circuitincludes first switching circuitry 905/1405/1905 to connect the set ofsignal lines to the plurality of pixels in the imaging mode anddisconnect the set of signal lines SL from the plurality of pixels 51 inthe depth mode. The second driving circuit includes second switchingcircuitry 910/1410/1910 to connect the set of signal lines SL to theplurality of pixels 51 in the depth mode and to disconnect the set ofsignal lines SL from the plurality of pixels in the imaging mode.

According to at least one example embodiment, each pixel 51 furthercomprises a second transfer transistor TG1 coupled to a second floatingdiffusion FD1 and the photoelectric conversion region PD.

According to at least one example embodiment, the second driving circuit815/1310/1810 applies a first transfer signal GD0 to the first transfertransistor TG0 of a first pixel during a first frame to generate a firstpixel value p0, applies a second transfer signal GD180 to the secondtransfer transistor TG1 of the first pixel during the first frame togenerate a second pixel p180 value, applies a third transfer signal GD90to the first transfer transistor TG0 of a second pixel during the firstframe to generate a third pixel value p90, and applies a fourth transfersignal GD270 to the second transfer transistor TG1 of the second pixelduring the first frame to generate a fourth pixel value p270 (see FIGS.11 and 12, for example). The first, second, third, and fourth pixelvalues are used to calculate a distance to an object.

According to at least one example embodiment, the first driving circuitcontrols the plurality of pixels to output color data for the colorimage in a second frame (see FIG. 12).

According to at least one example embodiment, the first, second, third,and fourth transfer signals have respective phase shifts of 0 degrees,180 degrees, 90 degrees, and 270 degrees compared to a driving signal ofa light source that emits light toward the object.

According to at least one example embodiment, the second driving circuitapplies the second transfer signal GD180 to the first transfertransistor TG0 of the first pixel during a second frame to generate afifth pixel value p180′, applies the first transfer signal GD0 to thesecond transfer transistor TG1 of the first pixel during the secondframe to generate a sixth pixel value p0′, applies the fourth transfersignal GD270 to the first transfer transistor TG0 of the second pixelduring the second frame to generate a seventh pixel value p2′70′, andapplies the third transfer signal GD90 to the second transfer transistorTG1 of the second pixel during the second frame to generate an eighthpixel value p90′ (see FIGS. 16 and 17).

According to at least one example embodiment, the first, second, third,fourth, fifth, sixth, seventh, and eighth pixel values are used tocancel fixed pattern noise in a distance calculation to the object (seeFIG. 25).

According to at least one example embodiment, the first driving circuitand the second driving circuit control the plurality of pixels through asame set of signal lines SL (see FIG. 18, for example).

According to at least one example embodiment, the first driving circuitcontrols the plurality of pixels to output color data for the colorimage in a third frame (see FIG. 17).

At least one example embodiment is directed to a system including alight source that emits infrared light, and an imaging device 1 thatincludes a pixel array including a plurality of pixels 51. Each pixel 51includes a photoelectric conversion region PD that converts incidentlight into electric charge, and a first transfer transistor TG0 coupledto a first floating diffusion FD0 and the photoelectric conversionregion PD. The imaging device 1 includes a first driving circuit tocontrol the plurality of pixels in an imaging mode to generate a colorimage based on visible light received from a scene, and a second drivingcircuit to control the plurality of pixels in a depth mode to generate adepth image based on the infrared light reflected from the scene.

At least one example embodiment is directed to a method that includesdriving, by a first driving circuit, a plurality of pixels in an imagingmode to generate a color image, and driving, by a second drivingcircuit, the plurality of pixels in a depth mode to generate a depthimage. The first driving circuit and the second driving circuit drivethe plurality of pixels through a same set of signal lines SL.

Any processing devices, control units, processing units, etc. discussedabove may correspond to one or many computer processing devices, such asa Field Programmable Gate Array (FPGA), an Application-SpecificIntegrated Circuit (ASIC), any other type of Integrated Circuit (IC)chip, a collection of IC chips, a microcontroller, a collection ofmicrocontrollers, a microprocessor, Central Processing Unit (CPU), adigital signal processor (DSP) or plurality of microprocessors that areconfigured to execute the instructions sets stored in memory.

As will be appreciated by one skilled in the art, aspects of the presentdisclosure may be illustrated and described herein in any of a number ofpatentable classes or context including any new and useful process,machine, manufacture, or composition of matter, or any new and usefulimprovement thereof. Accordingly, aspects of the present disclosure maybe implemented entirely hardware, entirely software (including firmware,resident software, micro-code, etc.) or combining software and hardwareimplementation that may all generally be referred to herein as a“circuit,” “module,” “component,” or “system.” Furthermore, aspects ofthe present disclosure may take the form of a computer program productembodied in one or more computer readable media having computer readableprogram code embodied thereon.

Any combination of one or more computer readable media may be utilized.The computer readable media may be a computer readable signal medium ora computer readable storage medium. A computer readable storage mediummay be, for example, but not limited to, an electronic, magnetic,optical, electromagnetic, or semiconductor system, apparatus, or device,or any suitable combination of the foregoing. More specific examples (anon-exhaustive list) of the computer readable storage medium wouldinclude the following: a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an appropriateoptical fiber with a repeater, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer readable signal medium may be transmitted usingany appropriate medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET,Python or the like, conventional procedural programming languages, suchas the “C” programming language, Visual Basic, Fortran 2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages such as Python, Ruby andGroovy, or other programming languages. The program code may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider) or in a cloud computing environment or offered as aservice such as a Software as a Service (SaaS).

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatuses(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable instruction executionapparatus, create a mechanism for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that when executed can direct a computer, otherprogrammable data processing apparatus, or other devices to function ina particular manner, such that the instructions when stored in thecomputer readable medium produce an article of manufacture includinginstructions which when executed, cause a computer to implement thefunction/act specified in the flowchart and/or block diagram block orblocks. The computer program instructions may also be loaded onto acomputer, other programmable instruction execution apparatus, or otherdevices to cause a series of operational steps to be performed on thecomputer, other programmable apparatuses or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

As used herein, the phrases “at least one,” “one or more,” “or,” and“and/or” are open-ended expressions that are both conjunctive anddisjunctive in operation. For example, each of the expressions “at leastone of A, B and C,” “at least one of A, B, or C,” “one or more of A, B,and C,” “one or more of A, B, or C,” “A, B, and/or C,” and “A, B, or C”means A alone, B alone, C alone, A and B together, A and C together, Band C together, or A, B and C together.

The term “a” or “an” entity refers to one or more of that entity. Assuch, the terms “a” (or “an”), “one or more” and “at least one” can beused interchangeably herein. It is also to be noted that the terms“comprising,” “including,” and “having” can be used interchangeably.

The foregoing discussion has been presented for purposes of illustrationand description. The foregoing is not intended to limit the disclosureto the form or forms disclosed herein. In the foregoing DetailedDescription for example, various features of the disclosure are groupedtogether in one or more aspects, embodiments, and/or configurations forthe purpose of streamlining the disclosure. The features of the aspects,embodiments, and/or configurations of the disclosure may be combined inalternate aspects, embodiments, and/or configurations other than thosediscussed above. This method of disclosure is not to be interpreted asreflecting an intention that the claims require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive aspects lie in less than all features of a singleforegoing disclosed aspect, embodiment, and/or configuration. Thus, thefollowing claims are hereby incorporated into this Detailed Description,with each claim standing on its own as an embodiment of the disclosure.

Moreover, though the description has included description of one or moreaspects, embodiments, and/or configurations and certain variations andmodifications, other variations, combinations, and modifications arewithin the scope of the disclosure, e.g., as may be within the skill andknowledge of those in the art, after understanding the presentdisclosure. It is intended to obtain rights which include alternativeaspects, embodiments, and/or configurations to the extent permitted,including alternate, interchangeable and/or equivalent structures,functions, ranges or steps to those claimed, whether or not suchalternate, interchangeable and/or equivalent structures, functions,ranges or steps are disclosed herein, and without intending to publiclydedicate any patentable subject matter.

Example embodiments may be configured according to the following:

(1) An imaging device, comprising:

a pixel array including a plurality of pixels, each pixel including:

-   -   a photoelectric conversion region that converts incident light        into electric charge; and    -   a first transfer transistor coupled to a first floating        diffusion and the photoelectric conversion region;

a first driving circuit to control the plurality of pixels in an imagingmode to generate a color image; and

a second driving circuit to control the plurality of pixels in a depthmode to generate a depth image.

(2) The imaging device of (1), further comprising:

a plurality of color filters that correspond to the plurality of pixels,wherein the plurality of color filters include red color filters, greencolor filters, blue color filters, and neutral color filters.

(3) The imaging device of one or more of (1) to (2), wherein the neutralcolor filters include white color filters, gray color filters, or blackcolor filters.(4) The imaging device of one or more of (1) to (3), further comprising:

an optical filter on the plurality of color filters and that passesvisible light and selected wavelengths of infrared light.

(5) The imaging device of one or more of (1) to (4), wherein the opticalfilter blocks wavelengths of light between a wavelength of the visiblelight and a wavelength of the selected wavelengths of infrared light.(6) The imaging device of one or more of (1) to (5), wherein the seconddriving circuit applies first, second, third, and fourth transfersignals to the first transfer transistor in first, second, third, andfourth frames, respectively, to generate a first pixel value for thefirst frame, a second pixel value for the second frame, a third pixelvalue for the third frame, and a fourth pixel value for the fourthframe, and

wherein the first, second, third, and fourth pixel values are used tocalculate a distance to an object.

(7) The imaging device of one or more of (1) to (6), wherein the first,second, third, and fourth transfer signals have respective phase shiftsof 0 degrees, 180 degrees, 90 degrees, and 270 degrees compared to adriving signal of a light source that emits light toward the object.(8) The imaging device of one or more of (1) to (7), wherein the firstdriving circuit controls the plurality of pixels to output color datafor the color image in a fifth frame.(9) The imaging device of one or more of (1) to (8), wherein the firstdriving circuit and the second driving circuit control the plurality ofpixels through a same set of signal lines.(10) The imaging device of one or more of (1) to (9), wherein the firstdriving circuit includes first switching circuitry to connect the set ofsignal lines to the plurality of pixels in the imaging mode anddisconnect the set of signal lines from the plurality of pixels in thedepth mode, and wherein the second driving circuit includes secondswitching circuitry to connect the set of signal lines to the pluralityof pixels in the depth mode and to disconnect the set of signal linesfrom the plurality of pixels in the imaging mode.(11) The imaging device of one or more of (1) to (10), wherein eachpixel further comprises:

a second transfer transistor coupled to a second floating diffusion andthe photoelectric conversion region.

(12) The imaging device of one or more of (1) to (11), wherein thesecond driving circuit applies a first transfer signal to the firsttransfer transistor of a first pixel during a first frame to generate afirst pixel value, applies a second transfer signal to the secondtransfer transistor of the first pixel during the first frame togenerate a second pixel value, applies a third transfer signal to thefirst transfer transistor of a second pixel during the first frame togenerate a third pixel value, and applies a fourth transfer signal tothe second transfer transistor of the second pixel during the firstframe to generate a fourth pixel value, and

wherein the first, second, third, and fourth pixel values are used tocalculate a distance to an object.

(13) The imaging device of one or more of (1) to (12), wherein the firstdriving circuit controls the plurality of pixels to output color datafor the color image in a second frame.(14) The imaging device of one or more of (1) to (13), wherein thefirst, second, third, and fourth transfer signals have respective phaseshifts of 0 degrees, 180 degrees, 90 degrees, and 270 degrees comparedto a driving signal of a light source that emits light toward theobject.(15) The imaging device of one or more of (1) to (14), wherein thesecond driving circuit applies the second transfer signal to the firsttransfer transistor of the first pixel during a second frame to generatea fifth pixel value, applies the first transfer signal to the secondtransfer transistor of the first pixel during the second frame togenerate a sixth pixel value, applies the fourth transfer signal to thefirst transfer transistor of the second pixel during the second frame togenerate a seventh pixel value, and applies the third transfer signal tothe second transfer transistor of the second pixel during the secondframe to generate an eighth pixel value.(16) The imaging device of one or more of (1) to (15), wherein thefirst, second, third, fourth, fifth, sixth, seventh, and eighth pixelvalues are used to cancel fixed pattern noise in a distance calculationto the object.(17) The imaging device of one or more of (1) to (16), wherein the firstdriving circuit and the second driving circuit control the plurality ofpixels through a same set of signal lines.(18) The imaging device of one or more of (1) to (17), wherein the firstdriving circuit controls the plurality of pixels to output color datafor the color image in a third frame.(19) A system, comprising:a light source that emits infrared light;an imaging device, comprising:

a pixel array including a plurality of pixels, each pixel including:

-   -   a photoelectric conversion region that converts incident light        into electric charge; and    -   a first transfer transistor coupled to a first floating        diffusion and the photoelectric conversion region;

a first driving circuit to control the plurality of pixels in an imagingmode to generate a color image based on visible light received from ascene; and

a second driving circuit to control the plurality of pixels in a depthmode to generate a depth image based on the infrared light reflectedfrom the scene.

(20) A method, comprising:

driving, by a first driving circuit, a plurality of pixels in an imagingmode to generate a color image;

driving, by a second driving circuit, the plurality of pixels in a depthmode to generate a depth image, wherein the first driving circuit andthe second driving circuit drive the plurality of pixels through a sameset of signal lines.

Any one or more of the aspects/embodiments as substantially disclosedherein.

Any one or more of the aspects/embodiments as substantially disclosedherein optionally in combination with any one or more otheraspects/embodiments as substantially disclosed herein.

One or more means adapted to perform any one or more of the aboveaspects/embodiments as substantially disclosed herein.

It is claimed:
 1. An imaging device, comprising: a pixel array includinga plurality of pixels, each pixel including: a photoelectric conversionregion that converts incident light into electric charge; and a firsttransfer transistor coupled to a first floating diffusion and thephotoelectric conversion region; a first driving circuit to control theplurality of pixels in an imaging mode to generate a color image; and asecond driving circuit to control the plurality of pixels in a depthmode to generate a depth image.
 2. The imaging device of claim 1,further comprising: a plurality of color filters that correspond to theplurality of pixels, wherein the plurality of color filters include redcolor filters, green color filters, blue color filters, and neutralcolor filters.
 3. The imaging device of claim 2, wherein the neutralcolor filters include white color filters, gray color filters, or blackcolor filters.
 4. The imaging device of claim 2, further comprising: anoptical filter on the plurality of color filters and that passes visiblelight and selected wavelengths of infrared light.
 5. The imaging deviceof claim 4, wherein the optical filter blocks wavelengths of lightbetween a wavelength of the visible light and a wavelength of theselected wavelengths of infrared light.
 6. The imaging device of claim1, wherein the second driving circuit applies first, second, third, andfourth transfer signals to the first transfer transistor in first,second, third, and fourth frames, respectively, to generate a firstpixel value for the first frame, a second pixel value for the secondframe, a third pixel value for the third frame, and a fourth pixel valuefor the fourth frame, and wherein the first, second, third, and fourthpixel values are used to calculate a distance to an object.
 7. Theimaging device of claim 6, wherein the first, second, third, and fourthtransfer signals have respective phase shifts of 0 degrees, 180 degrees,90 degrees, and 270 degrees compared to a driving signal of a lightsource that emits light toward the object.
 8. The imaging device ofclaim 6, wherein the first driving circuit controls the plurality ofpixels to output color data for the color image in a fifth frame.
 9. Theimaging device of claim 1, wherein the first driving circuit and thesecond driving circuit control the plurality of pixels through a sameset of signal lines.
 10. The imaging device of claim 9, wherein thefirst driving circuit includes first switching circuitry to connect theset of signal lines to the plurality of pixels in the imaging mode anddisconnect the set of signal lines from the plurality of pixels in thedepth mode, and wherein the second driving circuit includes secondswitching circuitry to connect the set of signal lines to the pluralityof pixels in the depth mode and to disconnect the set of signal linesfrom the plurality of pixels in the imaging mode.
 11. The imaging deviceof claim 1, wherein each pixel further comprises: a second transfertransistor coupled to a second floating diffusion and the photoelectricconversion region.
 12. The imaging device of claim 11, wherein thesecond driving circuit applies a first transfer signal to the firsttransfer transistor of a first pixel during a first frame to generate afirst pixel value, applies a second transfer signal to the secondtransfer transistor of the first pixel during the first frame togenerate a second pixel value, applies a third transfer signal to thefirst transfer transistor of a second pixel during the first frame togenerate a third pixel value, and applies a fourth transfer signal tothe second transfer transistor of the second pixel during the firstframe to generate a fourth pixel value, and wherein the first, second,third, and fourth pixel values are used to calculate a distance to anobject.
 13. The imaging device of claim 12, wherein the first drivingcircuit controls the plurality of pixels to output color data for thecolor image in a second frame.
 14. The imaging device of claim 12,wherein the first, second, third, and fourth transfer signals haverespective phase shifts of 0 degrees, 180 degrees, 90 degrees, and 270degrees compared to a driving signal of a light source that emits lighttoward the object.
 15. The imaging device of claim 14, wherein thesecond driving circuit applies the second transfer signal to the firsttransfer transistor of the first pixel during a second frame to generatea fifth pixel value, applies the first transfer signal to the secondtransfer transistor of the first pixel during the second frame togenerate a sixth pixel value, applies the fourth transfer signal to thefirst transfer transistor of the second pixel during the second frame togenerate a seventh pixel value, and applies the third transfer signal tothe second transfer transistor of the second pixel during the secondframe to generate an eighth pixel value.
 16. The imaging device of claim15, wherein the first, second, third, fourth, fifth, sixth, seventh, andeighth pixel values are used to cancel fixed pattern noise in a distancecalculation to the object.
 17. The imaging device of claim 15, whereinthe first driving circuit and the second driving circuit control theplurality of pixels through a same set of signal lines.
 18. The imagingdevice of claim 15, wherein the first driving circuit controls theplurality of pixels to output color data for the color image in a thirdframe.
 19. A system, comprising: a light source that emits infraredlight; an imaging device, comprising: a pixel array including aplurality of pixels, each pixel including: a photoelectric conversionregion that converts incident light into electric charge; and a firsttransfer transistor coupled to a first floating diffusion and thephotoelectric conversion region; a first driving circuit to control theplurality of pixels in an imaging mode to generate a color image basedon visible light received from a scene; and a second driving circuit tocontrol the plurality of pixels in a depth mode to generate a depthimage based on the infrared light reflected from the scene.
 20. Amethod, comprising: driving, by a first driving circuit, a plurality ofpixels in an imaging mode to generate a color image; driving, by asecond driving circuit, the plurality of pixels in a depth mode togenerate a depth image, wherein the first driving circuit and the seconddriving circuit drive the plurality of pixels through a same set ofsignal lines.